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METHOD AND APPARATUS FOR OPTIMIZING QUANTIZER 
METHOD AND APr^^ ^^^^^ ENCODER 

BACKGROUND OF THE INVENTION 

5 The tavention relates to computing quantization values used for encoding 

coeffients of a digital image or video frame and more particularly to optimizing 

the computed quantization values to reduce distortion in the digital image or video 
frame when encoding is performed with a limited number of bits. 

In many of today's image and video coders, the quality of encoded images 
1 0 is controlled by selecting one or more quality parameters. Block-based image and 
video coders, use a parameter known as a quantization scale or step for each block 
of pixels in the image. The quantization steps aie used for scaling pixel values 
within the same step ranges to the same values. Image blocks encoded w,tl, the 
same quantization scale have approximately the same quality. The number of b.ts 
needed for encoding an image depends on desired image quality (quantization 
scales) and onthe inherent statistics of the imag.. As a result, different images 
encoded with the same scales (same image quality) will occupy a different number 
ofbits. 

In many applications, the number ofbits available for encoding one or 
several frames is fixed in advance, and some technique is necessary to select the 
quantization scales that will produce that target number ofbits and encode the 
video frames with the highest possible quality. For example, in a digital video 
recording a group of frames (GOP) must occupy the same number ofbits for an 
efficient fast-forward/fast-rewind capability. In video telephony, the channel rate, 
25 communication delay, and size of encoder buffers determine the available number 
ofbits for one or more flumes. 

Existing quantizer or buffer control methods are classified into three major 
types A first type of quantizer control method encodes each image block several 
times with a set of quantization scales. The number ofbits produced for each case 
30 is measured and a scale for each block is smartly selected so the sum of the b.ts 
for all combined blocks hits the desired target bit number. The first type of 
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quantizer control techniques cannot be used for real-time encoding because of the 
high computational complexity required to encode each image block multiple 
times. 

The first type of quantizer control is described in the following 
5 publications: K. Ramchandran, A. Ortega, and M. Vetterli, "Bit Allocation for 
Dependent Quantization with Applications to Multi-Resolution and MPEG Video 
Coders," IEEE Trans, on Image Processing, Vol. 3, N. 5, pp. 533-545, September 
1994; W. Ding and B. Liu, "Rate Control of MPEG Video Coding and Recording 
by Rate-Quantization Modeling," IEEE Trans, on Circuits and Systems for Video 
1 0 Technology. Vol. 6, N. 1, pp. 12-19, February 1996; and L.J. Lin, A. Ortega, and 
C.C. J. Kuo, "Rate Control Using Spline-Interpolated R-D Characteristics," Proc. 
of SPIE Visual Communications and Image Processing, pp. Ill -122, Orlando, FL, 
March 1996. 

A second type of quantizer control technique measures the number of bits 
1 5 spent in previously encoded image blocks and measures other parameters such as, 
buffer fullness, block activity, etc. These measurements are used to select the 
quantization scale for the current block. The second type of quantizer control is 
popular for real-time encoding because of its low computational complexity! 
However, the second type of quantizer control is inaccurate in achieving the target 
20 number of bits and must be combined with additional encoding techniques to 
avoid bit or buffer overflow and underflow. 

The second method is described in the following publications: U.S. Patent 
No. 5,038,209 entitled "Adaptive Buffer/Quantizer Control for Transfonn Video 
Coders", issued August 6, 1991 to H.M. Ming; U.S. Patent No. 5,159,447 en titled 
25 "Buffer Control for Variable Bit-Rate Channel", issued October 27, 1992 to B.G. 
Haskell and A.R. Reibman; and U.S. Patent No. 5,141,383 entitled "Pseudo- 
Constant Bit Rate Video Coding with Quantization Parameter Adjustment", issued 
August 3 1 , 1 993 to C.T. Cheng and A.H. Wong. 

A third type of quantizer control technique uses a model to predict the 
30 number of bits needed for encoding the image blocks. The quantizer model 
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includes the blocks' quantization scales and other parameters, such as, block 
variances. The quantization scales are determined by some mathematical 
optimization of the encoder model. The third type of quantizer control is 
computationally simple and can be used in real-time, but is highly sensitive to 
5 model errors and often produces inaccurate results. 

The third type of quantizer control technique is described in the following 
pubHcations. E.D. Frimout, J. Biemond, and R.L. Lagendik, "Forward Rate 
Control for MPEG Recording." Proc. of SPIE Visual Communications and Image 
Processing, Cambridge, MA,.pp. 184-194. November 1993; U.S. Patent No. 
10 5 323,1 87 entitled, "Image Compression System by Setting Fixed Bit Rates". 

issued June 21, 1994 to K. Park and A. Nicoulin, "Composite Source Modeling 
for image Compression," Ph.D. Thesis N. 1 444 (1995), Ecole Polytechnigue., 
Federale de Lausanne, Chapter 6,1995. 

Thus, a need remains for improving the image quality of quantized image 
1 5 or video frames while reducing.the time and computational complexity required to 
generate optimized quantization values. 

SUMMARY OF THE INVENTION 
A quantizer controller generates quantization values using a new block- 
adaptive, Lagrangian optimization. The quantizer controller is updated and 
20 improved using information from earlier quantized blocks. The quantizer 

controller is robust to model errors and produces results as accurate as type-1 
quantizer control techniques, while having the simpler computational complexity 
of the type-2 quantizer control techniques. 

The quantizer controller identifies a target bit value equal to a total number 
25 of bits available for encoding tl.e fiame. A total amount of distortion in the frame is 
modeled according to the predicted quantization values assigned to each one of the 
blocks. The predicted quantization values are characterized according to an amount 
of energy in each block and a number of bits available for encoding each block. 
Optimum quantization values are adapted to each block by minimizing the modeled 
30 distortion in the frame subject to the constraint that the total number of bits for 
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encoding the frame is equal to the target bit value. Each block is then encoded with 
the optimized quantization value. 

The quantizer controller is adaptive to each block by reducing quantization 
values for the blocks having less energy and increasing the quantization values for 
the blocks having more energy. The quantization values assigned to the blocks are 
also optimized according to a number of image blocks remaining to be encoded and 
a number of bits still available for encoding the remaining image blocks. 

Different weighting factors are optionally applied to tlie quantization values 
that vary the accuracy of the encoded .blocks. One weighting factor is applied to the 
quantization values according to the location of the block in the frame. Optimized 
quantization values are applied to blocks in each frame, frames in a group of 
multiple frames or applied generally for any region in an array of image data. 

The quantizer controller only encodes the image once to accurately 
generate the quantization values for each block. The quantization values produce 
a target number of bits for the encoded image or video frame. Thus, the quantizer 
controller is less computationally exhaustive than a quantizer control technique of 
similar accuracy. 

The general framework of the quantizer controller can be used in a variety 
of quantizer/rate control strategies. For example, the quantizer controller can be 
used to select in real-time the value of the quantization scales for the Discrete 
Cosine Transform DCT-based encoding of the frame macroblocks in the current 
video coding standards MPEG 1-2 and 4, H.261, H.263, and H.263+. A frame, 
several frames, or several macroblocks within a frame are encoded with a fixed 
number of bits. 

The foregoing and other objects, features and advantages of the invention 
will become more readily apparent from the following detailed description of a 
preferred embodiment of the invention, which proceeds with reference to the 
•accompanying drawings. 
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BRIEF DESCRIPTION OF DRAWINGS 

FIG. 1 is a schematic diagram of muUiple image frames each including 
multiple blocks assigned optimized quantization values according to the invention. 
FIG. 2 is a block diagram of an image coder according to one embodiment 

5 of the invention. 

FIG. 3 is a step diagram for generating the optimized quantization values. 
FIGS. 4 and 5 show results from applying the optimized quantization 

values to image data. 

FIG. 6 is a block diagram of the quantizer controller according to one 

1 0 embodiment of the invention. 

DETAILED DESCRIPTION 

A block-based image coder 12 is used to describe the invention. HP^ever, 

the invention can be used for controlling the quantizer of any image or video 

coder. 

1 5 Referring to FIG. 1 , in block-based image coding, images 1 5 are 

transmitted in multiple frames 26. Each frame 26 is decomposed into multiple 
image blocks 14 of the same size, typically of 16x16 pixels per block. The 
number of bits B, produced after encoding an ith image block 14, is a function of 
the value of a quantization parameter and the statistics of the block. For 
20 example, image block i = 9 contains more image information (energy) a, than 

image block i = 17. This is because the image in block i = 9 contains portions of a 
facial image along with background infomiation. Conversely, image block i = 17 
has less image information energy a, because it contains substantially the same 
background imagery in substantially each pixel location. 
25 Referring to FIG. 2, the pixel values for each image block 14 are 

transformed into a set of coefficients, for example using a Discrete Cosine 
Transform (DCT) in block transfonn 16. These coefficients are quantized in block 
quantization 18 and encoded in coder 20. Bits B^of tlie encoded and quantized 
image blocks 14 are then transmitted over a communication channel 21 over a 
30 telephone line, microwave channel, etc. to a receiver (not shown). The receiver 
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includes a decoder that decodes the quantized bits and an inverse transform block 
that performs an Inverse Discrete Cosine Transform (IDCT). The decoded bits Bj 
are then displayed on a visual display screen to a user. 

Quantization of the transformed coefficients in quantization block 18 is a 
5 key procedure since it determines the quality with which the image block 14 will 
be encoded. The quantization of the ith block 14 is controlled by the parameter, 
Qs- In the H.261 and H.263 video coding standards, Q, is known as the 
quantization step for the ith block and its value corresponds to half the step size 
used for quantizing the transformed coefficients. In the MPEG-1 and MPEG-2 
1 0 standards, Q; is called the quantization scale and the jth coefficient of a block is 
quantized using a quantizer of step size Qj Wj , where Wj is the jth value of a 
quantization matrix chosen by the designer of the MPEG codec. 

Let N be the number of 16x1 6 image blocks in one image frame 26. The 
total number of bits B available for encoding one image frame 26 is: 

15 

B = B, +B2 + B3+ ... +Bn, (1) 

where the value of B depends on the quantization parameters selected, 
Qi. Q2. • • •» Qn. and the statistics of the blocks. The invention comprises a 

20 quantizer controller 22 (FIG. 1) that chooses optimum values for the Qj's for a 

limited total number of available bits B for encoding the frames 26. The quantizer 
controller 22 is implemented in a variety of different maps including in software 
in a programmable processing unit with dedicated hardware. 

In image coding, the image blocks 14 are said to be intracoded or of class 

25 intra. In video coding, many of the blocks 14 in a frame 26 are very similar to 
blocks in previous frames. The values of the pixels in a block 14 are often 
predicted from previously encoded blocks and only the difference or prediction 
error is encoded. These blocks are said to be intercoded or of class inter. The 
invention can be used in frames with both intra and inter blocks. 

30 

6 
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Encoder Model 

The following model in equation 2 identifies the number of bits invested in 
the ith image block : 

b,=a(k|v-c), (2) 

The value Q,is the quantizer step size or quantization scale, A is the number of 
pixelsinablock(e.g:,inMPEGandH.263 A = 16^ pixels), K and C are 
constants, and a, is the empirical standard deviation of the pixels in the block. 



10 



15 



20 



25 



The valueP,© is thejth pixel in the ith block and P, is the average of the pixel 
values in the block where, 

A .,=1 

For color images, the P,0) 's are the values of the luminance and 
chrominance components of the respective pixels. The model in equation 2 is 
derived using a rate-distortion analysis of the block's encoder. The value of K in 
equation 2 depends on the statistics of the image blocks 26 and the quantization 
matrix used in the encoder. For example, it can be shown that if the pixel values 
are approximately uncorrelated and Gaussian distributed, and the quantization 
matrix is flat with unitaiy weights (i.e., w, =1 for all j), then K=7r /ln2. The 
constant C in equation 2 models the average number of bits per pixel used for 
encoding the coder's overhead. For example, C accounts for header and syntax 
information, pixel color or cl^ominance components, transmitted Q values, 
motion vectors, etc. sent to the receiver for decoding the image blocks. If the 
values of K and C are not known, they are estimated with an inventive technique 

7 
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described below in the section entitled, "Updating the Parameters of the Encoder 
Model". 

Equation 5 models distortion D for the N encoded blocks. 



5 where the a ,'s are weights chosen to incorporate the importance or cost of the 
block distortion. For example, larger a .'s are chosen for image blocks having 
artifacts more visible to the human eye or for image blocks that belong to more 
important objects. in the scene. If a , -a 2 f= • ■ • =a n = 1 , the distortion 
represented by equation 5 is approximately the mean squared error (MSE) 
1 0 between the original and encoded blocks. 
Optimization 

The quantizer controller 22 (FIG. 1) selects the optimal quantization 
values, Q*„ Q*2, ... , Q*n, that minimize tlie distortion model in equation 5, 
subject to the constraint that the total number of bits must be equal to B as defined 
1 5 in equation 1, which can be expressed mathematically as follows: 



20 of Lagrange is used to convert the constrained-minimization in equation (6a) to tlie 



(5) 



(6a) 




The next objective is to find a formula for each of the Q' 's. To do this, the method 



"followingf 




(6b) 
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wh«e X is called ft= Lagrange multiplier. Next, equation (2) is used for B, m (6b) 



to obtain: 

q;,-..,q; = 



argmin - 
Q,.->Qn ^ 



Finally, by setting partial derivatives in (6c) to zero, the following formula is 
5 derived for the optimal quantizer step size for the i-th image block: 

(6) 



Moreover, if i-1 blocks 26 have already been quantized and encoded, the^optimal 
quantization parameter for the ith block is, 



10 



-iJ(B,-ANiC)a,^- 



(7) 



where = N-i+l is the number of image blocks that remain to be encoded and 
B , is the number of bits available to encode them, 



15 



B,=B 



-Eb3=b,_.-b,.=b>_.-a(k^^c], 



(8) 



where B„ is obtained using equation 2 with the optimized quantization value Ql, . 
Thus, equations 6 and 7 generate optimized quantization values that minimize 
20 distortion for a limited number of available bits. As a result, using the same 

number of bits, the image in frame 26 in FIG. 1 will have less distortion than other 
quantization schemes when displayed on a display unit at the receiver end of the 
channel 21. 
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QUANTIZER CONTROL METHOD 

FIG. 3 describes the steps performed by quantizer controller 22 (FIG. 2) 
for selecting quantizer values used for encoding N image blocks 14 with B bits. 
5 Note that N could be the number of blocks in an image, part of an image, several 
images, or generally any region of an image. 
Step 1. Receive energy values and initialization. 

Pixel values for the N image blocks are obtained to the quantizer controller 
22 frorn the digital image (FIG. 2) in step 1 A. Jnitialization is performed in step ■ 
10 IB by setting i = 1 (first block), B, = B (available bits), N, = N (number of 
blocks). Let S, = Za^a^ , where the 's are found using equation 3 and the 
a, 's are preset (e.g., set a , =a ^ = . . . =a ^ = 1 to minimize mean squared error). 
In one example, the amount of energy a,, is derived from the DCT coefficients of 
the pixel values generated by transform block 16. 
15 For a fixed mode, the values of the parameters K and C in the encoder 

model in equation 7 are known or estimated in advance. For example, using hnear 
regression, K, = K and C, = C. For an adaptive mode, the model parameters are 
not known, K, and C, are then set to some small non-negative values. For 
example, experiments have shown K, = 0.5 and C, = 0 to be good initial 
20 estimates. In video coding, K, and C, can be set to the values Kn+, and Cn^., , 
respectively, from the previous encoded frame. 

Step 2. Compute the optimal quantization parameter for the ith block. 

If the values of the Q-parameters are restricted to a fixed set (e.g., in 
QP=Q/2 and talces v^ues 1,2,3...., 31,), Q^* is rounded to the nearest value 
25 in the set. The square root operation is then implemented using look-up tables, 
where 



■ V(§i-AN..Ci)a, ' 
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Step 3. Encode the ith block with a block-based coder. 

B[ is the number of bits used to encode the ith block, compute 

5 Bi„=Bi-B;, Si.,= S,-a..a,. andN,„=Ni-l. 

Step 4- Update quantizer values. 

In step 4, the parameters K,,, and C,., are updated in the quantizer 
controller 22. For the fixed mode, IC,., = K, C,, = C. For the adaptive mode, the 
1 0 updates K,,, and Q,, are found using a model fitting technique. One example of a 
model fitting technique is described below in tlie section entitled "Updating^ 
Parameters in the Quantizer controller". 
Step 5. Generate quantizer value for next block. 

If i = N in decision step 5, quantization values have been derived for all image 
1 5 blocks 14 and the quantizer controller 22 stops. If all of the image blocks 14.have 
not been quantized, the quantizer controller 22 receives the coefficients for the 
next image block i = i+1 in step 6, and jumps back to step 2. The quantization 
value for block i = i+1 are then derived as described above. 

Referring to FIGS. 4 and 5, the frames of video sequences encoded by 
20 quantizer controller 22 where compared to those of a Telenor H.263 offline 

method, which is the quantizer control technique adopted for MPEG-4 anchors. 

In FIG. 4, the total number of bits per video frame obtained by the 
quantization technique described in FIG. 3 are shown in sohd line. The H.263 
offline encoding technique is shown in dashed line. Encoding was perfonned on 
25 133 frames of a well-known video sequence "Foreman". The target number of 
bits B is 11200 bits per frame. FIG. 5 is like FIG. 4, but for 140 frames of the 
video sequence "Mother and Daughter" with B=6400. 

The quantizer controller 22 produces a significantly more accurate and 
steady number of bits per frame. Similar results were obtained for a wide range of 
30 bit rates. In the experiments, there were little if no visible differences in the 

quaUty of the two encoded video sequences. The signal to noise ratio performance 
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of the images processed by quantizer controller 22 was only 0.1-0.3 dB lower on 
average. Thus, even though the image is only encoded once, quantizer controller 
22 achieves the target bit rate accurately with high image quality at every frame. 
Alternative Implementations 
5 Several quantization variations are based on the base quantization 

optimization framework discussed above. If the computation of all 's in Step 
IB of FIG. 3 cannot be performed in advance, a good estimate for S, is used, such 
as the value of S, from the previous video frame 26. 

A low-complexity estimate of S , can be used, in order to further reduce 
1 0 computational complexity. For the low complexity estimate, equation 3 is 
replaced by equation 9, 



■■^tabs(P,U)-P,), (9) 



1 5 where abs(x) is the absolute value of x. In video coding, the mean value of pixels 
in inter blocks is usually zero and hence equation 9 may be simplified by setting 
^=0. 

A fixed optimization selects the quantization parameters using equation 6 
instead of equation 7. To do this, in step 3 in FIG. 3, the values for Bj^, , Sj^, , and 
20 Nj+i are replaced by B;^, = B, Si^, = S,, and N;^, = N , respectively. 

For a variable-rate channel, if the number of bits available after encoding i 
blocks changes to B , because of a change of channel rate or other factors, set 
Bj ~,'^Bln"step'3. 

The quantization model defined in equation 2 can be generalized to 
25 equation 10, 

a.* 

B.=A.(K^ + C). (10) 
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Where A, is the number of pixels in the jth region. The region of quantization does 
not need to be a block. Additional model parameters .|> and y can either be set 
prior to quantization or obtained using parameter estimation techniques described 
below. If the quantization model in equation 1 0 is used in step 2, the optimized 
quantization values Q* 's are derived using equation 11, 



10 



15 



q; = 



(11) 



^B,-Ci;A„ot^- 
step 1, S, isreplacedby 5, = i(A„a,f )-^a.r • ^stepS. S,, isreplaced 
5,-(A,a*r*^a7•^^ 



by 



Encoding Intra and Inter Blocks. 

If some of the blocks to be encoded are of class intra (in the same frame) 
and some inter (between different frames), performance of the quantizer controller 
22 can be improved by dividing the standard deviation of the intra blocks by a 
factor # . Specifically, after computing the value for the a/s in step 1 , the 
factor VP is applied as follows: 

if kth block is intra 
otherwise 



The factor p is. 



20 The values K, and K, are the averages of the K's measured for the intra and inter 
blocks, respectively. The value of (3 is estimated and updated during encoding. 



During experimentation it was 



fovmd that using a constant P = 3 works well. 
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Frame-Based Quantizer Control. 

If the quantization step is fixed for all the blocks 14 within a frame 26, the 
same quantizer controller 22 shown in FIG. 1 can be used for encoding one or 
several frames. The parameters are reinterpreted so that: N = Number of frames, 
5 B = Bits available for encoding the N frames, i = Frame number in the video 
sequence, Qj = Quantization step for all the blocks in the ith frame, and A = 
Number of pixels in a frame. 

The parameters a, ,a. ,B; , are the weight, variance, and bits for the ith 
frame, respectively. The parameters Kj and C; are updates of the coder model for 
1 0 that frame. 

If computational complexity is not an issue, each image block 14 can be 
encoded several times and, using a classical model fitting procedure (e.g., least- 
squares fit, linear regression, etc.), a good estimate of tlie K^'s and C/s for the 
blocks can be found in advance. Then, in step 2, the quantization values Q* are 
1 5 determined according to. 



the updating and computation of this model parameter. In that case, observe that 
Q* in step 2 is simply. 




In Step 1, S, is replaced by S^ = Z^lK^Cj^a,^ , and in step 3, Sj^, is replaced by 



20 S,^, = - Jk^c 



iCTja; . To reduce the complexity, one can set C=Ci=0 and avoid 




or equivalently. 
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15 



Q* = 1^ , where Ai= A and T = -^^B, 



Any subset of the different techniques described in "Alternative Implementations" 
can be combined and used together. 
Updating Parameters of the Encoder Model. 
5 The following is one technique for updating the parameters Kj+, and Ci+, in 

the quantizer controller 22. This update technique is used with the adaptive mode 
described above in step 4. Other classical parameter estimation or model fitting 
techniques such as least squares, recursive least squares, Kalman prediction, etc. 
can alternatively be used. The model parameters can be updated in every block, 
1 0 frame, group of blocks, or group of frames. 

The model parameters in one embodiment of the invention, are upd to ror 
estimated on a block-by-block basis using the following weighted averages. 



^' Aaf ■ A 'Q*f 



The values of K and C predict B\ using equation 2. Alternatively, in some codecs 
these formulas are used for measuring K, and C, : 

where B ocT.i is the number of bits spent for the DCT coefficients of the i-th image 
20 block. 

The averages of the Kj'j and C^^ are computed as follows. 
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The updates are a linearly weighted average of K, , Cj and their respective initial 
estimates K„ C„ 



If the general model in equation 10 is used, a variety of estimators can be used to 
estimate the additional parameters (f) and y . These parameters can also be 
updated on a block-by-block basis, and the ith updates <t), and y , can be found 
using averaging techniques similar to those for K,, Cj . 

1 0 Selection of the a,. Weights. 

The a j values can be chosen to incorporate the importance or weight of 
block distortions. If default values are used a , =a 2 = • • ■ =«- n = 1, the MSB 
distortion is minimized between the original and the encoded blocks. Otherwise, 
the MSE distortion decreases in blocks with larger a j's and increases where the 
15 a i's are smaller. Two examples are described below for choosing the a ; weights. 
A region, such as a rectangular window in a video telephone image, is assigned a 
larger value of a ; and, in tum, smaller quantization values. The weighted region 
will be coded with better quality since a smaller quantization scale is used to 
quantize pixel values. 

20 People usually pay more attention to the central region of a picture. Thus, 

larger values of a ; are assijgned to the regions near the center of the picture. A 
pyramid .formuia.is..used-to-assign-larger-values-of-a-i-to-blocks-Gloser-to the center 
of the frame. Specifically, let Bx and By be the number of blocks along the 
horizontal and vertical coordinates, respectively. The weight for the i-th block is 

25 computed as follows. 
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where (a, + a,) and a, are the height and offset of the pyramid, respectively, and 
and by are the horizontal and vertical position of the block in the fi-ame. For 
example, choosing a.=15 and a,=l causes the a , value of the center block to be 16 
5 times that of boundary blocks. 
Block Joining 

In a codec, the values of the quantizer values used Q*„ . . ., Q*n (see step 2 
above) need to be encoded and sent to the decoder. For example, in H.263. 
quantizer values are encoded in a raster-scan order and there is a five-bit penalty 

1 0 for changing the quantizer value. At high bit rates, the bit overhead for changing 
the quantizer values is negligible and the optimization techniques described above 
are effective. However, at very-low bit rates, this overhead is significant and;, 
some technique is needed for constraining the number of times that the quantizer 
changes. Unfortunately, existing optimization methods that take quantization 

1 5 overhead into account are mathematically inaccurate or computationally 
expensive. 

In another aspect of the invention, a heuristic method joins blocks of 
similar standard deviation together into a set so that the quantizer value remains 
constant within the set. This technique is referred to as block joining, and reduces 
20 the changes of the quantizer at lower bit rates. Block joining is accomplished by 
choosing the values of the a, weights as follows, 

2^(l-a,.)-^a, ^<0.5^ 
1 otherwise 

25 where B/(AN) is the bit rate in bits per pixel for the current frame. The values of 
B, A, and N were defined earher as the number of bits available, number of pixels 
in a block, and number of blocks, respectively. If the bit rate is above 0.5, the a, 's 
are all equal to 1 and hence have no effect. At lower bit rates, the a, 's linearly 
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approach the respective ct, 's and progressively reduce the range of the Q*'s. In 
fact, if the bit rate is 0, then a, = a, and all the quantizer values are equal, and 
hence all blocks are joined into one set, 

FIG. 6 is a detailed block diagram of the quantizer controller 22 shown in 
FIG. 2. The quantizer controller 22 in one embodiment is implemented in a 
1 0 general purpose programmable processor. The functional blocks in FIG. 6 
represent the primary operations performed by the processor. Initialization 
parameters in block 3 1 are either derived from pre-processing the current image or 
from parameters previously derived from previous frames or from prestored values 
in processor memory (not shown). Initialization parameters include N„ S„ B,, K, 

1 5 and C, (or Kn+, and Cn+, from a previous frame). 

The image is decomposed into N image blocks 14 (FIG. 2) of A pixels in 
block 30. The energy of the pixels in each block is computed in block 32, The 
weight factors assigned to each block are computed in block 34. The amount of 
energy left in the image is updated in block 40 and the bits left for encoding the 
20 image are updated in block 38. Parameters for the encoder model are updated in 
block 42 and the number of blocks remaining to be encoded are tracked in block 
44. The processor in block 36 then computes the optimized quantizer step size 
according to the values derived in blocks 32, 34, 40, 38, 42 and 44. 

Having described and illustrated the principles of the invention in a 

2 5 preferred embodiment thereof; it should be apparent that the invention can be 

modified in arrangement and detail witliout departing from such principles. I 
claim all modifications and variation coming within the spirit and scope of the 
following claims. 
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CLAIMS 

1 . A method for encoding multiple blocks in a frame of image data, 
comprising: 

5 identifying a target bit value equal to a total number of bits available for 

encoding the frame; 

predicting a total distortion in the frame according to quantization values 
assigned to each one of the blocks, the quantization values characterized according 
to an amount of energy in each block and a number of bits assigned to each block; 
1 0 adapting optimum quantization values for each of the multiple blocks by 

minimizing the total predicted distortion in the frame subject to a constraint that the 
total number of bits available for encoding the frame is equal to the target bit^value; 
and 

encoding the blocks with the predicted optimum quantization values. 

15 

2. A method according to claim 1 wherem the optimum quantization 
values are generated using a Lagrange optimization on the predicted total distortion. 

3. A method according to claim 1 wherein tlie optimum quantization 
20 values are derived according to the following. 



( AK a , ^ 

Q'=J(B-ANc)^r;S"'"' • 

where, Q; * is the optimum quantization value for each block i, N is the number of 
25 blocks in the frame, B is the total number of bits available for encoding the frame, 
A is a number of pixels in each of the multiple blocks, K and C are constants 
associated with the image blocks, a, is an empirical standard deviation of pixel 
values in the block, and a ; is a weighting incorporating the importance of 
distortion in the block. 

19 
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4. A method according to claim 1 including adjusting the optimum 
quantization values according to a number of image blocks remaining to be encoded 
and a number of bits still available for encoding the remaining image blocks. 

5 

5. A method according to claim 3 including using a K parameter and a 
C parameter on a block-by-block basis to adjust the optimum quantization values for 
each of the multiple blocks, the K parameter modeling correlation statistics of the 
pixels in the image blocks and the C parameter modeling bits required to code 

1 0 overhead data. 



6. A method according to claim 5 including deriving the optimum 
quantization values in either a fixed mode where the K and C parameters are known 
in advance or an adaptive mode where the K and C parameters are derived according 

15 to the K and C parameters of previously encoded blocks. 

7. A method according to claim 6 wherein the adaptive mode includes 
the following st^s: 

deriving values for the K and C parameters that exactly predict the number 
20 of bits B used for encoding previous blocks; 

deriving averages for the derived K and C parameters for the previously 
encoded video blocks; and 

predicting the K and C parameters for a next video block by linearly 
weighting the average K and C parameters according to the initial estimates for the 
25 K and C parameters. 

8. A method according to claim 7 wherein the values of K and C that 
predict B are. 



20 
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5 where B'dctj is a number of bits spent for the DCT coefficients of the current 
image block; 

the averages of K and C are; 



10 



K. =— K , +-Ki and C; = -^C,_, + tC; ; and 



the linearly weighted averages of K and C are, 

i N - i i ~ N -i ^ 



15 



9. A method according to claim 3 wherein the amount of energy in tlie 
frame is not determined in advance and is estimated according to the following 
where. 



^' ^(B,-ANiCi)a, 

20 where, 

and the 's and ct^ 's are those obtained for the blocks in a previously encoded 
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10. A method according to claim 9 including encoding the image blocks 

several times to estimate parameters K,, K^, . . . K„ and C„ Cj C„ for each of the 

image blocks and then deriving a super optimum quantization value by setting: 



|(s,-a|c,)«' 

N . 

wherci S\ = Sv^**'*"* ^i+i ~ ^\ - 



11. A method according to claim 1 including predicting optimum 
1 0 quantization values for each frame according to tlie following steps: 

identifying a multiframe bit value equal to a total number of bits available 
for encoding multiple frames; 

modeling a total amount of distortion in the multiple frames according to 
quantization values assigned to each one of the frames, the quantization values 
1 5 characterized according to an amount of energy in each fi-ame and a number of bits 
assigned to each frame; 

predicting optimum quantization values for each frame that minimize the 
total modeled distortion in the multiple frames; and 

encoding each frame with the predicted optimum quantization value. 

20 

12. A method according to claim 1 including applying weighting factors 
to each of the optimum-quantization -values according to location of the blocks in the 
frame. 

25 13. A method according to claim 1 including controlling a number of 

different optimum quantization values assignable to the blocks by assigning the 
same quantization values to blocks having similar standard deviation values. 



22 



wo 98/35500 



PCT/US98/01827 



13. A method according to claim 1 including controlling a number of 
different optimum quantization values assignable to the blocks by assigning the 
same quantization values to blocks having similar standard deviation values. 

14. A method according to claim 1 3 wherein the optimum quantization 
values are controlled by assigning the following weighting values. 



1 otherwise 



1 0 where B/(AN) is the bit rate in bits per pixel for a current frame, B is the number of 
bits available, A is a number of pixels in the block, N is the total number of blocks 
in the frame, and ct,- is the standard deviation for the pixels in the blocks. 

15. A method for quantizing regions in a video image, comprising: 
1 5 receiving image information for different regions; 

predicting an amount of distortion created in the video image according to 
quantization values assigned to the regions, the predicted distortion characterized 
according to the amount of information in the region and the number of bits 
available for encoding the information in the regions into the quantization values; 
20 optimizing the quantization values assigned to the regions so that the amount 

of predicted distortion in the regions is minimized for the number of available bits; 
and 

encoding the regions with the optimized quantization values. 

25 16. A method according to claim 1 5 wherein the optimized quantization 

values are derived as follows. 
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where, 

-2- Jv_ 

5 

y , ^ , K and C are constants, A; is a number of pixels in an ith region, a,, represents 
energy of the pixel values for the ith region, Bj represents the number of bits 
1 0 available and a j is a weighting factor incorporating importance of the region 
distortion. 



1 7. A method according to claim 1 5 wherein the optimized quantization 
values for a selected region is derived as follows: 

1 5 summing the energy in each of the regions to determine a total energy in tlie 

video image; 

multiplying the total energy with an amount of energy in the selected region; 
scaling the multiplied energies according to a scaling factor; and 
squaring the scaled energies thereby generating the optimized quantization 
20 value for the selected region. 

18. A method according to claim 1 7 wherein scaling the multiplied 
energies includes the following: 

applying a first scaling factor proportional to the number of regions in the 

2 5 frame remaining to be quantized; and 
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applying a second scaling factor that varies for each region according to a 
total number of bits available for encoding the frame and a total number of bits 
already used to encode previous regions in the frame. 

5 19. A method according to claim 1 8 including applying a third and 

fourth scaling factor, the third scaling factor modeling correlation statistics in the 
region and the fourth scaling factor representing overhead data in the encoded frame. 

20. A method according to claim 1 9 including applying a fifth scaling 
1 0 factor proportional to a number of pixels in each region. 

21 . A method according to claim 1 5 wherein the energy in each regipn.is 
proportional to a standard deviation for pixel values or a sum of the absolute values 
of the pixels in relation to an average of all pixel values in the same region, 

15 

22. A method according to claim 1 5 including predicting the total energy 
in the video image by taking the total energy for a previous video image. 

23 . A method according to claim 1 5 wherein the predicted optimum 
20 quantization values are reduced for the blocks having less energy and the predicted 

optimum quantization values are increased for the blocks having more energy. 

24. A method according to claim 21 wherein the energy is adjusted by a 
scaling value for the pixels in intracoded regions, the scaling value characterized 

2 5 according to different K values representing correlation statistics for different types 
of coded regions. 

25. An encoder for quantizing regions in video frames, comprising: 

a circuit for detecting an amount of video information in one of the regions; 
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a quantizer controller assigning quantization values to each region that 
minimize an amount of predicted distortion in the video frames for a target bit value, 
the quantizer controller predicting an amount of distortion created in the video 
frames before the information in the region is actually quantized and adjusting the 
5 quantization values assigned to each region to minimize the predicted distortion 

according to a constraint that a total number of available bits for encoding the fi-ames 
is equal to the target bit value; and 

a quantizer quantizing the video information in the regions according to the 
adapted quantization values associated with the regions generated from the quantizer 
1 0 controller. 

26. An encoder according to claim 25 including a transfomi circuit 
receiving the video image at an input and generating transform coefficients at an 
output, the quantizer quantizing tlie transform coefficients for each region according 
15 to the associated quantization values. 



wo 98/35500 



PCT/US98/01827 



1 /6 



14- 



i=9- 



i=17— 




FIG.1 



FRAME N 



FRAME 



FRAME 1 



SUBSTITUTE SHEET (RULE 26) 



wo 98/35500 



PCT/US98/01827 



2 /6 



O 



C/) 




1— 








m 




en" 






on 




LiJ 

o 




o 




o 


_1 




< 






t 






H- 


z 


CL 


o 


O 






























< 








o 





OU- 
CD< 



o 
o 

_J 

_ o 
< 



<|jj 



CM 



o 

<C/) 

am 
o 













to , fO 










CM , CM 










I 6" 











SUBSTITUTE SHEET (RULE 26) 



wo 98/35500 



PCT/US98/01827 



3 /6 



22 



RECEIVE PIXEL 
ENERGY VALUES 



-1A 



INITIALIZE 
QUANTIZER 



-IB 



COMPUTE OPTIMAL 
QUANTIZATION 
VALUE Qi* FOR 
BLOCK i 



ENCODE BLOCK i 



UPDATE QUANTIZATION 
PARAMETERS 




YES 



RECEIVE COEFFICIENTS 
FOR BLOCK i = i +1 



FIG.3 



HNfsnnnin; <wo sb35Sooai 



SUBSTITUTE SHEET (RULE 26) 



wo 98/35500 



PCT/US98/01827 



4 /6 




ZCM CO CD CM CO CO CN 

^ ^ O O CD O 



SDOCIO: <WO 983S500A1_I_5 



SUBSTITUTE SHEET (RULE 26) 



wo 98/35500 



PCT/US98/01827 



5 /6 




SUBSTITUTE SHEET (RULE 26) 



wo 98/35500 



6 /6 



PCT/US98/01827 



z m 



oa 

? < 



o 
(/)0 

OUI LU 
Z)<0 

<z 

□_UJ 











ot- .. 


1 


0_Q 


— 


R OF B 
REMA] 
ENCOD 


II 

+ 
1 


111 1— 


z 
II 


31- 


z 


Z 





OLiJ 

a. CO 

UJUJ 

I- 

OZ5 



>-o 



cr 



uj o D 



1 • — I ■ - 

3Q-0 O 

o 



o .1 



CD 

LJOZQD 
I-O ,1 
<Z " 



CM 



OO o 

Oti. 



UJ< I 
UJ^ I 



Q.llJ(/) 

II 



o 



< 
q: 
u. 

o 

UJ 

o 

> 

cr 
o 

UJ 

o 
< 



< 



S o 

CO 

o 



2 

- 3 

z °= 

o ^ 

»- 2 

< 3 

M z 

II 



^ CD 
^ U. 



< 

O 

I— 
II 

L 





UJ 
















u. 














UJ 




o 


> 


< 


UJ 




a: 




a. 


UJ 




X 


o 


f— 




cr 


U. 


o 




u. 


+ 


Ui 


z 


_l 


o 


CD 




ILA 


+ 
z 


< 




AV 


TO 


CO 




»— 


< 


CD 


ID 


U. 


C3 
UJ 


O 




cr 


;oR 


MBi 


o 


NU 


Il_ 
c5" 


1— 


Q 


UI 

o 


z 


< 


cr 




< 


in 




d 


II 


ll_ 










« 


«■ 



UJ>^_I 
OOoUJ 



_9e35500A1_l_> 



SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 


j International application .No. 
PCT/US98/01827 


A. CLASSIFICATION OF SUBJECT MATTER 
IPC(6) :H04N 7/12 

A^cordulg to International. Patent Giassification (IPC) or to both national classification and IPC 


R FIELDS SEARCHED ■ 

Minimum documentation searched (classification system followed by classification symbols) 
U.S. : Please See Extra Sheet. , 


Documentation searched other than minimum documentation to the extent that such documenU are mcluded m the fields searched 


Electronic data base consulted during the intemational search (name of daU base and. where practicable, search terms used) 


n nnrilMENTS CONSIDERED TO BE RELEVANT 




Category* 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


X 
X 
X 
Y 


US 5,144,423 A (KNAUER ET AL) 01 SEPTEMBER 1992, FIGS. 
1-7 AND 9-17. 

US 5,134,475 A (JOHNSTON ET AL) 28 JULY 1992, FIGS. 1-7 
AND 9-17. 

US 5,475,501 A (YAGASAKI) 12 DECEMBER 1995, FIGS. 1 
AND 2. 


1, 4, 11-13, 15, 
AND 17-26 

1, 4, 11-13, 15, 
AND 17426 

1, 15, AND 25 


4, 11-13, 17-23, 
AND 26 


Fx] Furt 


her documents are listed in the continuation of Box C. □ See patent family annex. 


-— 7~^~rZ 1 l-wr docirniont publiihed .fior the inU.mation.1 niinR ddto or priority 

Speoni c«uigon« of cilad dociiin«itt: ^ conHict wi«h Ihe application but cited to undentand 

•A- dooumenl deHning Um goneiml itata of Iha art which ii not eoniidorad the principle or theory underlying the inY ontion 

,0 b« of p«ticular r.l»«.c. ^ document of particular relev anoe. the cUimad invntion ouutot b. 

•E" eaiHer documant pubtithad on or after the intemauonal filing dale coniidered novel or cannot be coniidered to involve an inventive itap 
.... . .,1. • when the document it taken alone 

•L' document which may throw doubu on priority claun(i) or which n 

cited 10 etubliih the publication date of another citation or other document of particular relevance; the claimed invention oannot be 
apecial reason (ai tpeoified) coniidered 10 involve an invenUve itep when the dooiunent ii 

•O- doeumeot referring to an oral di.clo.ure. o.e, exhibiuon or other b^g'obJiou'i'to"a pe "on .killed in the art 

•p- doeuroent publiihed prior 10 the international filing dale but later than document member of the .ame potent family 


Date of the actual completion of the intemational search 
27 APRIL 1998 


Date of mailing of the intemational search report 

1 6 JUL 199 8 


Name and mailing address of the ISA/US 
Commissioner of Patents and Trademarks 
Box POT 

Washington, D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized officer /^^^^^^^ ^_ U/u^ 

TOMMY CHIN ^ 
Telephone No. (703)305-4715 



Form PCT/ISA/210 (second sheetXJuly 1992)* 



INTERNATIONAL SEARCH REPORT 



Interaational application No. 
PCT/US98/01827 



C (Continuation). DOCUMENTS CONSIDERED TO BE RELEVANT 



Category' 



Citation of document, with indication, where appropriate, of tlie relevant passages 



Relevant to claim No. 



X,P 
Y,P 

X,P 
Y,P 



US 5,374,958 A (YANAGIHARA) 20 DECEMBER 1994 FIGS 
11-14. 



US 5.426,463 A (REININGER ET AL) 20 JUNE 1995, FIGS. 1-3 
7, AND 8. 



US 5,627,581 A (KONDO) 06 MAY 1997, FIGS. 1-3 5 10-12 
14. AND 15. 



US 5,606,371 A (KLEIN GUNNEWIEK ET AL) 25 FEBRUARY 
1997, FIGS. I, 2, AND 4-6. 



US 5,550,590 A (SAKAZAWA ET AL) 27 AUGUST 1996, FIGS. 
1, 2, 7, 8, AND 12. 

US 5,734,755 A (RAMCHANDRAN ET AL) 31 MARCH 1998, 
FIGS. 1, 5, 6, 8, AND 9. 



1, 15, AND 25 



4, 11-13. 17-23, 
AND 26 



1, 15, AND 25 



4, 11-13, 17-24, 
AND 26 

, 15, AND 25 



4, 11-13. 17-24. 
AND 26 

1, 15, AND 25 



4, 11-13, 17-24, 
AND 26 



Form PCT/ISA/210 (continuation of second sheetXJuly 1992)* 



INTERNATIONAL SEARCH REPORT 



Inlemalional application No. 
PCTAJS98/01827 



B. FIELDS SEARCHED 
Minimum documentation searched 
Classiflcation System: U.S. 

348/405. 384, 390, 400. 401. 402. 403. 404, 409. 411, 412. 415. 4 19. 420. 421 
IPC{6): H04N 7/12 



Fonii PCT/ISA/210 (extra sheetXJuly 1992)* 

<wo saassooAi j > 



